Peer Review Workshop

Introductions and Objectives

Show and tell

• Here’s POSEIDON application to the DTS fishery
• We would like to know:
  • Is anything fundamentally wrong or missing?
  • If you could direct additional work, what would you add or change?
  • How would you score the current work so far?
• About:
  • The assumptions that run the model
  • What data we used and how
  • The match between the California Current and our simulation

A tale of two reviews

• January: POSEIDON in theory
• September: POSEIDON in practice
• How flexible can POSEIDON be?

Others

• Two great models of groundfish on the west coast:
  • Kaplan, Holland and Fulton (2014)
  • Toft, Punt and Little (2011)
• We are different because:
  • Middle complexity
  • Agents are fishers rather than port groups
  • Fit against EDC data
  • Endogenous discovery and ITQ prices
  • Behavioural testing

Spoilers

• The good:
  • We calibrated the model to aggregate empirical observations
  • Adaptive agents outperform both perfect and statistically fit agents
  • Error is insensitive to:
    • Shocks to calibrated parameters
    • Large changes to the biological rules
    • Fish movement rate
• The bad less good:
  • Model allocates too much profits and quotas to South Oregon and North California
  • Error is sensitive to:
    • Changes in fish distribution
    • Market micro-structure

POSEIDON overview

What is POSEIDON

• POSEIDON is a fishery agent-based model
• POSEIDON assumes fishers are simple and adaptive to:
  • changes in the biological layer
  • changes in policy
  • changes in market conditions

January throwback

POSEIDON in September

• The challenge is to go from a theoretical model to an applied one
• Much work went into making the biology layer more realistic
• Very little was changed in the behavioural component of the fleet

Biology and Geography

DTS Species

• We model 5 species of fish:
  • Dover Sole
  • Shortspine Thornyhead
  • Longspine Thornyhead
  • Sablefish
  • Yelloweye Rockfish

Biological Rules

• POSEIDON tracks number of fish by age and sex
• Recruitment, Spawning and Natural mortality are:
  • Global
  • Yearly
  • Deterministic
  • Copied from stock assessment models
• Fishing mortality is event-driven
• Movement is turned off
• No predator-prey interaction

Biological Rules - Steve

The world

The world - 2

• Gridded Map:
  • 50x120 cells
  • 15.15km each side
• Each cell contains a set of abundance vectors \(\Big\{ A_{1,\text{male}},A_{1,\text{female}},A_{2,\text{male}},\dots,A_{5,\text{female}} \Big\}\) where: \[ A_{i,\text{male}} = \begin{bmatrix} \text{No. of male fish of species } i \text{ of age } 0\\ \text{No. of male fish of species } i \text{ of age } 1\\ \vdots\\ \text{No. of male fish of species } i \text{ of max age }\\ \end{bmatrix}\]

Distributing Fish

• From stock assessment outputs we have global abundance vectors
• We need to parcelize them into cell-wise abundance vectors
  • Use Essential Fish Habitat (NMFS, 2013) model to weigh each cell of the map
  • Normalize weights
  • Allocate fish per cell as % of abundance predicted by weights
• Yearly new recruits allocated through same proportion

Statistical Areas

Statistical Areas -2

• We group multiple map-cells in the same “statistical area”
  • Make logit regressions possible
  • Used by some heuristics
• They have no intrising biological property

Fleet Dynamics

Fleet

• 90 Boats
  • Using 2010 DTS census from EDC
  • Homogeneous by quota and gear
  • From Seattle to Monterey

Gear

• Given:
  • \(q_i\): catchability
  • \(s_i\): selectivity
  • \(r_i\): retention
  • \(A_i\): cell abundance for species \(i\)
• A boat fishing for one hour catches a set of vectors \(x_1,\dots,x_5\) where: \[ x_i = q_i \left( s_i \circ r_i \circ A_i \right) \]
• \[ \begin{bmatrix} \text{No. of fish caught of species } i \text{ of age } 0\\ \text{No. of fish caught of species } i \text{ of age } 1\\ \vdots\\ \text{No. of fish caught of species } i \text{ of max age }\\ \end{bmatrix} = q_i \cdot \begin{bmatrix} r_{i,0}s_{i,0} A_{i,0}\\ r_{i,1}s_{i,1} A_{i,1}\\ \vdots\\ r_{i,\text{max}}s_{i,\text{max}} A_{i,\text{max}}\\ \end{bmatrix} \]

Physical Capacity

• Boats have an upper limit of 170 days at sea (Lian et al, 2009)
• We removed 50 days spent fishing shrimps off-model
• DTS boats can at most fish for 120 days

Regulations

• RCA
  • 200 m depth contour
  • Fixed
• ITQ:
  • Quota spread uniformly
  • Perfect matching
  • No geographical separation
• Bi-monthly limits (pre 2011):
  • Modelled as untradeable IQ
  • Homogeneous
  • Higher than 2011 quotas
  • Discard allowed

Costs

• Vessel move at 16kph and consume 3.54 l/km gasoline
• Fuel costs changes yearly
• On top of fuel expenditures, each hour out costs 165$
• Fishers making two years of consecutive losses quit the fishery
  • They retain quotas however and lease them for cash

Decision Rules

• Each trip, agents fish in one cell
• Different candidate heuristics choose where to fish each trip
  • Which one fits fishing patterns the best?

Options, options

• Statistical Agents:
  • Logit
  • Locked
• Adaptive Agents:
  • Bandit
  • Explore-Exploit-Imitate
  • Heatmapping
  • Social Annealing
• Other:
  • Random
  • Perfect

Logit Agents

• Agents use fit discrete choice models to make decisions
  • Obtain 2011-2014 logit parameters
  • Simulate agents between 2011-2014 using those parameters
  • Make decisions in terms of statistical areas
• Habit issues

Statistically locked agents

• Agents go to statistical areas in proportion to how often they were visited in real life
  
• Useful for calibration

“Perfect” Agents

• Agents know the \(\Pi\) they would make in any statistical area
• Probability of going to a statistical area \(i\): \[ \frac{e^{\Pi_i}}{\sum_j e^{\Pi_j}} \]
• Common in many bio-economic models (Kaplan for example)

Epsilon-greedy bandit agents

• Stochastic agent:
  • With probability \(\epsilon\) : pick a statistical area at random
  • Else go to the statistical area with highest average reward so far
• Suffers from bad explorations very far from port

Explore-Exploit-Imitate

• Stochastic agent:
  • With probability \(\epsilon\) : pick a cell at random around where you last fished
  • Else if your “friends” are doing better, go where they are going
  • Else go back to where you last fished
• Each agent has 2 “friends”

Heatmapping agent

Heatmapping agent - 2

• Draws a mental map of where he thinks fish is
• Stochastic agent:
  • With probability \(\epsilon\) : pick a cell at random around where you last fished
  • Else go where the the mental heatmap is highest

Social Annealing agents

• Fisher goes back to the same spot if making more than \(k\)% of coast-wide average income
• Otherwise fisher explore

Agents recap

• Statistical Agents:
  • Logit
  • Locked
• Adaptive Agents:
  • Bandit
  • Explore-Exploit-Imitate
  • Heatmapping
  • Social Annealing
• Other:
  • Random
  • Perfect

Data

Data - How

• Biological Data
  • Biomass and Abundance: input
  • Fish location: input
• Human Data
  • Regulations and Constraints: input
  • Fishing Decisions: output
  • Fishing Outcomes: output

Data - Where

• Biological Data
  • Biomass and Abundance: Stock Assessment
  • Fish location: EFH
• Human Data
  • Regulations and Constraints: PFMC - NOAA
  • Fishing Decisions: Logbook Data
  • Fishing Outcomes: EDC

Fishing Outcomes

Fishing Decisions

• Indirect Inference
• Logbook fits

Fishing Decisions - Steve

Calibration

What’s calibration anyway?

• Model fitting
• Only a few free parameters
  • Catchability per species
  • Average hold size
  • Behavioural Parameters
• We want to change them such that model is as close as possible to real data

Error measurement

• Aggregate all fishing outcomes into a single error number
  1. Take \(y_i\) as average of observed fishing outcome \(i\), with standard deviation \(\sigma_i\)
  2. Take \(\hat y_i\) as simulated fishing outcome
  3. Fishing outcome error is: \[ \text{Error} = \sum \frac{| \hat y_i - y_i |}{\sigma_i} \]
• Aggregate all logbook decisions into a single error number
  1. Take \(\beta_i\) as the parameter of logit fit to logbook, with standard error \(\sigma_i\)
  2. Take \(\hat \beta_i\) as the parameter of logit fit to simulated logbook
  3. Logbook error is: \[ \text{Error} = \sum \frac{| \hat \beta_i - \beta_i |}{\sigma_i} \]

How to

• Fitting in two steps:
  1. Using statistically locked agents, change catchabiity and hold size to minimize distance between simulated Fishing Outcomes and empirical ones
  2. Fixing catchability and hold size, change behavioural parameters to minimize distance to logbook fits
• Judging behaviours:
  1. Run fully calibrated model, rank decision rules by fishing outcomes
  2. Run fully calibrated model at different starting date, rank decision rules by fishing outcomes

Step 1 - Dashboard

Step 2 - Dashboard

Error considerations

• Aggregation issues
• Step 2 calibration doesn’t directly target dashboard “looks”
• We focused on average error

Ranking

Results

1. Random agents perform very poorly (Janssen trap avoided)
2. Adaptive agents degrade better than statistical agents
3. Explore Exploit Imitate work well even uncalibrated
4. Adaptive agents fit better than perfect agents

What’s wrong with perfect?

What’s wrong with perfect? - 2

Validation

Validation and Sensitivity Analysis

• What is validation?
• Try to predict 2015
• Look at other variables we didn’t calibrate for
  1. Quota prices
  2. Profit Trends
  3. Active Fishers
  4. Geographical distribution of effort

Leave one out validation

• We calibrated the model on 2011-2014 data
• Can we predict what happens in 2015?
  • Landings + Economic Data (no logbook)
  • Boring

Validation Error

The importance of weights - 1

The importance of weights - 2

The importance of weights - 3

The importance of weights - 4

Look at intermediate output

Quota Price - Sablefish

Quota Price - Yelloweye

Quota Price - Shortspine

Quota Price - Longspine

Quota Price - Dover Sole

Quota Price - Summary

• Decent Fits
• Only partially informative

Profit Trends

Active Fishers

Geographical Allocation

Sensitivity Analysis

• How sensitive are the results to the following:
  • Small shocks to calibrated parameters
  • The biological model used
  • The distribution of fish
  • Fish Movement
  • Trading Rules

Sensitivity Analysis - Spoilers

• How sensitive are the results to the following:
  • Small shocks to calibrated parameters - Insensitive
  • The biological model used - Insensitive
  • The distribution of fish - Sensitive
  • Fish Movement - Insensitive
  • Trading Rules - Sensitive

Active Nonlinear Tests

• Allow all parameters to change 10% around their calibrated value
• Search (by optimiser) for the parameters that maximise the error
• 

Step 1 - ANT

• 10% in parameters increase errors by 9%

Step 2 (EEI) - ANT

• No meaningful error difference

Step 1 and 2 (EEI) - ANT

• No meaningful error difference

Step 1 and 2 (Heatmap) - ANT

Biology Sensitivity Analysis

• Switch biology layer:
  • Biomass-based (not age structured)
  • Deriso-Schnute recruitment
  • Different initial biomasses
• Recalibrate catchabilities and hold-size

Biology Sensitivity Analysis - Ranking

Swapping Maps

• Keeping biomass and processes constant
• Stop allocating fish as EFH maps predict
• Allocate fish by CPUE map
• Apply linear interpolation

Swapping maps

Errors

What is happening

• Adaptive rules have same performance as perfect knowledge
• Smooth surface too easy
  • Lesson learned: if the problem is easy, assuming adaptive agents is the same as assuming perfect knowledge

What is happening - proof

Why shouldn’t we assume this is a better map?

How does the model react?

Northern Oregon advantage

Movement Sensitivity

Trading Sensitivity

Discussions